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ABSTRACT 

With the advancements in 
the computing power of personal 
computers, pc-based flight 
simulators and trainers have 
opened new avenues in the 
training of airplane pilots. It 
may be desirable to have the 
flight simulator make a 
quantitative evaluation of the 
progress of a pilot's training 
thereby reducing the physical 
requirement of the flight 
instructor who must, in turn, 
watch every flight. 

In an experiment, 

University students conducted 
six different flights, each 
consisting of two level turns. 
The flights were three minutes 
in duration. By evaluating 
videotapes, two certified flight 
instructors provided separate 
letter grades for each turn. 
These level turns were also 
evaluated using two other 
computer based grading methods . 
One method determined automated 
grades based on prescribed 
tolerances in bank angle, 
airspeed and altitude. The other 
method used was deviations in 
altitude and bank angle for 
performance index and 
performance grades . 

INTRODUCTION 


With the invention of faster 
personal computers it has become 
possible to use flight 
simulation on cost effective 
computers . Because the 
advantages of training on flight 
simulators includes savings in 
time and money, computer based 
training provides researchers 
with increased opportunity to 
investigate such factors that 
may affect flying and pilot 
training. Investigation of 
scoring practices is needed to 
evaluate different strategies in 
training. Computerized scoring 
is desirable as it could, if 
properly implemented, give 
immediate feedback about the 
pilot's performance. 

Straight and level flight, 
climb, descent and level turn 
are regarded as the four basic 
flight maneuvers in a Federal 
Aviation Administration (1995) 
publication on private pilot 
practical test standards. For 
evaluation of a trainee's 
flight, it prescribes certain 
tolerances in the basic flight 
parameters. Following FAA's 
guidelines, an expert flight 
instructor directly evaluates 
the performance of a trainee's 
flight . 

For modern training 
simulators, Vreuls and Obermayer 
(1985) emphasize the need of 
automated performance measures 



as a substitute for the 
evaluation of performance by 
direct observations. They 
present several benefits of the 
automated performance measures. 
The authors have suggested 
several methods for validating 
automated measures . One method 
includes the need of experts to 
judge performance quality, and 
then determine which measures 
correlate with the experts' 
judgments . 

Williams (2000) used grade 
points for flying performances 
on the simulator. To validate 
the automated scoring, several 
flights flown on a simulator 
were simultaneously evaluated by 
the computer and by a certified 
flight instructor. He found 
acceptable correlation for 
straight and level flights but 
low correlation for climb, 
descent and turning maneuvers. 
Ali, Khan, Rossi, Crane, 
Guckenburger and Bageon (2001) 
proposed that a performance 
measure which represents an 
increase or decrease of 
performance at different stages 
of training is a valid measure 
to assess progress in training 
even if the measure is not 
adopted for certifying a trained 
pilot . 

We used an RMS value of 
deviation in bank angle and 
altitude as a performance index. 
We compared the automated 
grades, and performance index 
with the instructors' letter 
grades. Following Williams 
(2000) method, we used grade 
points for our automated grade. 
Furthermore, two instructors 
independently evaluated the 
flights and gave letter grades. 

In an attempt to create a 
numerical grade using the RMS 
value, performance index and 
automated grades were graphed 
and a regression line was drawn. 


University students performed 
six different flights. Each 
flight consisted of two level 
turns. For a level turn task, 
two kinds of performance 
measures were devised, automated 
grades and performance index. 

In the automated grades, A, B, C 
and D grades were based on 
prescribed tolerances in bank 
angle, and altitude. A 
performance index was based on 
deviations in bank angle and 
altitude. Video recordings were 
made of the HUD and relevant 
instrument panel gauges . Two 
certified flight instructors 
independently evaluated the 
videos of the flights. This 
study attempts to correlate the 
performance index, the automated 
grades and instructors' 
evaluations . 

EQUIPMENT 

The experiment was 
performed in the Flight Vehicle 
Design Lab at Tuskegee 
University. Flights were 
conducted on a training 
simulator that had LiteFlite 
version 3.3 installed. The PCs 
for the simulator were two Heavy 
Metal computers made by Quantum 
3D. Each computer has 2 
processors running at 400Mhz, 
400MB RAM, and has three extra 
video cards for the Out of the 
Window (OTW) view. Three 
display monitors showed the OTW 
view. The center monitor 
displayed not only a forward 
view, but also a heads up 
display (HUD) . Figure 1 shows 
the entire setup including the 
four monitors and a moving map 
display monitor on the right of 
the picture . The moving map 
display was not used in this 
experiment . Figure 2 shows a 
close up of the inside the 
cockpit view screen. The 
joystick was a Saitek X36F and 
throttle was Saitek X35T 
controller. Rudder pedals used 
were from CH products. Williams 



(2000) gave more details on the 
fidelity of the equipment used. 




Figure 1 : Simulator setup 


Figure 2 . Instrument Panel 
(Inside the cockpit view) 


METHOD 

Level turn task 

This task was to be a level 
turn from 0 degrees to 180 
degrees heading. The pilot, in 
this scenario, was to bank the 
airplane 30 degrees to the right 


or left while flying at a speed 
of 90 knots. After the plane 
reached 180 degrees, the pilot 
was to bank in the same or 
opposite direction at a new bank 
angle of 45 degrees. The pilot 
was to hold the bank angle and 
speed until the airplane reached 
360 degrees. After that time 
the pilot was to fly straight 
and level until the simulator 
stopped. 

Each turn task was recorded 
by videotaping the HUD on the 
center monitor and the turn and 
bank instruments on the inside 
the cockpit monitor. 


Instructors' grades 

Two Certified Flight 
Instructors were then asked to 
view the videotaped sessions and 
write down a score (A,B,C,D,E or 
F) for, in turn, altitude, bank 
angle and airspeed over the time 
slots: 21-72 seconds, and 
approximately 120 - 171 seconds. 
These time intervals represented 
each of the two turns in a 
flight. An average of the 
three -parameter grades was 
calculated to represent the 
instructors' grade for a single 
turn. This was used as a data 
point. For the six flights 
there were 12 data points 
representing all turns. 

Automated grades 

The automated grade 
criteria were based on the 
requirements of 90 knots +3knots 
in airspeed, 30 or 45 degrees ± 
3degrees in bank angle and 10000 
ft ± 50 feet in altitude. Flight 
parameters within these limits 
were graded as 'A' . Deviations 
of + 6 knots in airspeed, + 6 
degrees in bank angle and + 100 
feet in altitude were graded as 
'B' . Deviations of ±9 knots in 
airspeed, +9 knots in bank angle 
and +150 knots in altitude were 
graded as 'C' . Deviations of +12 
knots in airspeed, ±12 degrees 
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in bank angle and +200 feet in 
altitude were graded as 'D' . 

And, deviations of < - 12 knots 
and > +12knots in airspeed, -12 
degrees and >+12 degrees in bank 
angle and < -200 feet and > +200 
feet in altitude were graded as 
'F' . The flight parameters were 
grading every three seconds . An 
average grade was then 
calculated for each of the 
turns . 

Performance Index and 
Performance Grades 

The performance index of the 
level turn flights were obtained 
by taking the root of the 
squared sum of the average 
deviations of the heading, 
altitude, and airspeed. 

It should be noted that as a 
performance index (P.I.) 
increases, the performance of a 
pilot actually decreases. For 
that reason, in order to obtain 
some reasonable 4 point scale 
score we used an equation of the 
form A/ (B+P . I . ) and the values 
tried for A and B were such that 
B= 0 . 25*A. 

RESULTS AND DISCUSSION 

Our set of evaluation 
consisted of 12 data points 
representing 6 flights with 2 
turns in each flight. Five such 
sets of evaluation were: 

Automated grades, instructor 1 
and instructor 2 grades, 
performance index and 
performance grade. A comparison 
of the different evaluations 
that were made is as follows. 


expected, as the subjects were 
novice pilots. However, the two 
instructors' scores were 
somewhat inconsistent with one 
another. In general, 
instructor#l graded higher in 
comparison with the automated 
grade, while the grading of 
instructor#2 was lower than the 
automated grade . 



Figure 3 . Scatter plot of 
Instructor One Grade vs. 
Automated grade (Turn Task) 



Automated grade 


Automated Grade versus 
Instructors' grades 

The comparison of the overall 
automated grade vs . the 
instructors' grades for each of 
the turns is shown as scatter 
charts in Figs. 3 and 4. As can 
be seen from the scatter plots, 
most of the scores are in the 
2.0 or below range. This was 


Figure 4 . Scatter plot of 
Instructor Two Grade vs. 
Computer Automated Grade (Turn 
Tasks) 
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Performance Index 
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Figure 5. Scatter plot of 
Automated Grade (AG) vs . 
Performance Index (PI) (Turn 
Task) 


Figure 7 . Scatter plot of 
Instructor #2 Grade vs. 
Performance Grade (Turn Task) 


There exists a negative 
correlation of the automated 
score with the performance 
index. This should be expected 
as the performance index has an 
inverse relationship with both 
the instructor grades and the 
automated score. However, the 
best-fit regression more 
resembles a curve of the form: 

AG = 0.5265/ > / A (-1.0198) (Figure 
5) . 



Figure 6. Scatter plot of 
Instructor #1 Grade vs. 
Performance Grade (Turn Task) 



Automated Grade 


Figure 8. Scatter plot of 
Performance Grade vs . Automated 
Grade (Turn Task) 

In order to obtain some 
reasonable 4 point scale score 
we used an equation of the form 
A/(B+P.I.) and the values tried 
for A and B were such that B= 

0 ,25*A. 

The formula: 1.0/ (0.25 + P.I.) 
seems to be better suited for 
the purpose of determining 
performance grade from 
performance index. This formula 
was used after several attempts 
with other similar formulae. 

Most of the P.I. values seemed 
to match with 0.25. 


The performance grades appear to 
correlate well with the 
automated grades and the 
automated grades correlate well 
with the averaged instructor 
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grades. The instructor #l's 
grades correlate better with the 
performance grade than does 
instructor #2 's grades. In 
general instructor #2's grades 
don' t seem to correlate well 
with any of our computer grading 
measures . 

This analysis suggests 
that letter-grades may not be 
appropriate for tracking 
training progress as for novices 
the improvements are 
incremental . Thus they may 
exhibit improvements in 
controlling individual flight 
parameters but still the overall 
'letter' grade may not show an 
improvement in overall skill . 
Although the correlations are 
moderate to high between 
instructors, the instructors 
both seem to have a systematic 
difference between their score 
and the automated grade . 

Further, the differences in 
correlations on different 
parameters between instructors 
and the computer grade suggests 
that perhaps, the instructors 
are making evaluations based on 
different factors from each 
other. Thus the factors 
influencing the assessments of 
the instructors need to be 
understood. Future studies could 
systematically address these 
issues . 

A failed attempt was made for a 
power regression between the two 
instructors grades and the 
performance index. 

An increase in performance 
grades for a level turn 
corresponds with an increase in 
performance for a novice pilot. 
Therefore, they can be 
considered for further study for 
the evaluation of progress in 
training of level turns in 
simulators . 


CONCLUSION 

Four kinds of grading have 
been obtained and compared for 
evaluating level turn flights 
performed by novice pilots in a 
simulator. They are: 
instructor's grades, automated 
grades, performance index and 
performance grades. 

Instructor's grades are the 
grades provided by a certified 
flight instructor. Automated 
grades are based on prescribed 
tolerances in bank angle, 
altitude and airspeed. The 
performance index (P.I.) is 
created using the square root of 
the sum of the squares of 
dimensionless deviations in 
altitude and bank angle. 
Performance grade, which is 
defined as 1/(0. 25 + P.I.), 
provides a reasonable 4 -point 
grading scale. The comparison 
of performance grades with the 
other three performance 
measures, leads to the 
suggestion that they deserve 
further study for the evaluation 
of progress in training of level 
turns in simulators. 
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